Skip to content

feat: replace KEYS() by SCAN()#122

Merged
robertsLando merged 1 commit into
masterfrom
keys-to-scan
May 20, 2025
Merged

feat: replace KEYS() by SCAN()#122
robertsLando merged 1 commit into
masterfrom
keys-to-scan

Conversation

@seriousme

Copy link
Copy Markdown
Contributor

This PR implements #119.

Few notable things:

  • scanBuffer (for cluster) only returns keys and needs to be executed on each individual cluster node
  • hscanBuffer (for non-cluster) always returns both keys and values
  • I have used generators to avoid buffering too much in memory.
  • non-wildcard and wildcard patterns can overlap (e.g. /hello/world and /hello/world/#) so we need to track packets (using the sentTopics Set) to avoid duplicates in the stream.

Kind regards,
Hans

@robertsLando robertsLando changed the title feature: replace KEYS() by SCAN() feat: replace KEYS() by SCAN() May 19, 2025

@robertsLando robertsLando left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Well done 😄

@robertsLando

Copy link
Copy Markdown
Member

@seriousme Out of curiosity did you also did a quick benchmark?

@seriousme

Copy link
Copy Markdown
Contributor Author

@seriousme Out of curiosity did you also did a quick benchmark?

No I didn't. The challenge for me is to decide what the test should look like.
E.g. a test with a single retained topic (or a few) might be great as a microbenchmark but not really predictive for real life trafic.
Similarly:

  • how many subscriptions does a single client on average use and how many of those contain wildcards?
  • is the average retained topic tree narrow and deep, or shallow and wide
  • how many topics are on average in the retained topic tree (with few topics , Keys() might be faster)
  • do we expect a few clients or a few thousand clients in parallel (resulting in resource contention in Redis)
  • should we test on single nodejs proces, multiple nodejs processes on the same host, multiple on multiple hosts
  • single node redis instance, redis cluster on a single node, redis cluster on multiple nodes.

So all in all there are so many variables involved that I doubt we can make a generic statement on "performance" 😉
The most likely answer will be 'it depends' however for certain uses cases (e.g. topics without wildcards) there can be serious benefits as dicussed in #122 .

If you are unsure whether we will break performance, we could make it an option if you want.
Or stay with the existing Keys() solution as apparently no one complains about the performance of that.

It was fun to solve this puzzle.

Kind regards,
Hans
ps. I know that it's probably not possible but some statistics on how people actually use Aedes would be nice to guide these kinds of decisions.

@robertsLando

Copy link
Copy Markdown
Member

@seriousme no worries I agree, I think anyway this is much better compared to the solution used before as keys() shouldn't be used on production as stated in redis docs so let's merge this and thanks 🙏🏼

@robertsLando robertsLando merged commit 71a5d1f into master May 20, 2025
6 checks passed
@robertsLando robertsLando deleted the keys-to-scan branch May 20, 2025 11:43
@seriousme

Copy link
Copy Markdown
Contributor Author

@robertsLando
While explaining the code to a friend I noticed that a small bug got in:

function wildCardPosition (pattern) {
const oneIndex = pattern.indexOf(qlobberOpts.wildcard_one)
const someIndex = pattern.indexOf(qlobberOpts.wildcard_some)
if (oneIndex > someIndex) {
return oneIndex
}
return someIndex
}

This should give the smallest value for wildcard position, but now it gives the biggest.

e.g. hello/+/world/# now gives 14 (the # ) where it should have returned 6 (the + )

I'll fix it tonight and try to add a test to prevent regression in the future.

Kind regards,
Hans

@robertsLando

Copy link
Copy Markdown
Member

Oh I totally missed that! My fault, I think adding some tests with different patterns would be safer yeah

@seriousme

Copy link
Copy Markdown
Contributor Author

The fix is in #123, matching retained messages lives in abstract.js, so I will try to add a pattern there.
Unfortunately that means having to release all 3 again , sorry :-(

@robertsLando

Copy link
Copy Markdown
Member

No worries :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants